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NThe  present  publication  describes  the  evaluation,  in  terms  of  speed  and  accuracy, 
of  four  configurations  of  procedures  for  inputting  information  into  a  semi-automated 
information  processing  system.  Sixty  USMA  Prep  School  enlisted  men  were  given  an  ex¬ 
perimental  task  requiring  each  to  translate  55  free-text  messages  into  computer-accept¬ 
able  terminology.  Accuracy  and  speed  of  two  input  procedures  were  each  compared  under 
two  conditions  of  verification.  In  one  procedure,  the  subjects  translated  the  incoming 
message  onto  a  paper  format  before  transcribing  on  a  CRT  screen  (off-line).  In  the 
other  procedure,  the  message  was  transcribed  directly  on  the  CRT  screen  (on-line).  In 
the  unverified  condition,  one  man  performed  the  input  operation  without  error  check;  in 
the  verified  condition,  two  men  translated  the  same  message,  compared  their  transla¬ 
tions,  and  resolved  differences  before  entering  the  information  into  the  data  base. 
Performance  results  under  the  four  experimental  conditions  were  also  compared  with  a 
similar  7th  Army  TOS  procedure  in  which  a  message  is  translated  onto  a  paper  format  and 
the  unverified  message  is  copied  on  the  CRT  screen  by  the  UIOD  (user  input-output 
device)  operator. 

In  the  present  experiment,  data  input  accuracy  was  significantly  increased  when 
free-text  messages  were  translated  directly  on  the  CRT  screen  rather  than  first  filled 
out  on  paper  formats  (11.2$  error  vs  14.8$).  Input  speed  was  practically  the  same 
under  both  methods.  When  two  operators  checked  tach  other's  translation  before  input¬ 
ting  to  the  data  base,  error  was  reduced  by  one-third  (10.5$  vs  I 5 . T$ ) >  hut  the  proce¬ 
dure  required  one-third  more  time  (6.81  min  vs  4.98  min).  Either  procedure  was  an  im¬ 
provement  in  accuracy  over  the  work  method  used  in  the  TOS.  The  present  study  strongly 
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suggests  that  incoming  messages  should  be  translated  directly  on  the  CRT 
screen.  Direct  CRT  input  wouM  reduce  error  while  eliminating  paper 
formats  and  need  for  UIOD  ope •  *’■  c  transcription.  Findings  further  sug¬ 
gest  that,  when  time  and  personnel  permit,  messages  should  be  verified 
for  consistency  before  entering  the  information  into  the  data  base. 
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FOREWORD 


Technological  advancements  have  led  to  increased  speed,  mobility,  and  destructive 
power  of  military  operations.  To  permit  commanders  to  make  tactical  decisions  consis¬ 
tent  with  rapid  change  and  succession  of  events,  information  on  military  operations 
must  be  processed  and  used  more  effectively  than  ever  before.  To  meet  this  need,  the 
Army  is  developing  automated  systems  for  receipt,  processing,  storage,  retrieval,  and 
display  of  different  types  and  vast  amounts  of  military  data.  There  is  a  concomitant 
requirement  for  research  to  determine  how  human  abilities  can  be  utilized  to  enable 
command  information  processing  systems  to  function  with  maximum  effectiveness. 

BESRL's  manned  systems  research  in  this  area  is  directed  toward  the  enhancement 
of  human  performance  and  facilitation  of  man-machine  interaction  in  relation  to  total 
system  effectiveness.  It  involves  experimentation  with  various  configurations  of  sys¬ 
tem  components,  considering  interactions  and  tradeoffs.  The  end  products-immediate 
or  ultimate--are  scientific  findings  on  human  capabilities  under  varying  conditions 
within  the  system.  The  findings  have  implications  for  systems  design,  development, 
and  operational  use.  The  present  publication  describes  the  evaluation,  in  terms  of 
speed  and  accuracy,  of  four  configurations  of  procedures  for  inputting  information 
into  a  semi-automated  information  processing  system. 

The  entire  research  effort  is  responsive  to  requirements  of  RDT&E  Project 
2GQ24701 A723.  "Human  Performance  in  Military  Systems.”  FY  1971  Work  Program,  and 
to  special  requirements  of  the  Assistant  Chief  of  Staff  for  Force  Development,  the 
Assistant  Chief  of  Staff  for  Intelligence,  the  U.  S.  Army  Combat  Developments  Command, 
and  the  U.  S.  Army  Computer  Systems  Command. 
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EVALUATION  OF  MAN-COMPUTER  INPUT  TECHNIQUES  FOR  MILITARY 
INFORMATION  SYSTEMS 

BRIEF 


Requirement: 

To  evaluate  alternative  procedures  for  preparation  and  input  of  information  into  an 
Army  tactical  operations  system. 


Procedure: 

The  accuracy  and  speed  of  two  input  procedures  were  each  compared  under  two  con¬ 
ditions  of  verification.  In  one  procedure,  the  incoming  message  is  translated  onto  a 
paper  format  before  being  transcribed  on  a  CRT  screen  (off-line).  In  the  other,  the 
message  is  translated  directly  on  the  CRT  screen  (on-line).  In  the  unverified  condition, 
one  man  performs  the  input  operation  without  error  check;  in  the  verified  condition,  two 
men  translate  the  same  message  and  compare  their  translations  before  entering  the  in¬ 
formation  into  the  data  base.  Results  under  the  four  experimental  conditions  were  also 
compared  with  a  procedure  similar  to  that  used  in  the  7th  Army  TOS  in  which  a  message 
is  translated  onto  a  paper  format  arid  the  unverified  message  is  copied  on  the  CRT  screen 
by  the  operator  of  a  user  input  output  device  (UIOD) .  Subjects  were  60  enlisted  men 
studying  at  the  USMA  Prep  School,  who  were  divided  into  four  groups  and  assigned  to 
the  four  experimental  conditions  so  as  to  furnish  data  for  analysis  of  variance. 


Findings: 

There  were  significantly  fewer  errors  when  the  message  was  input  directly  on  the 
CRT  than  when  paper  formats  were  used  as  an  intermediate  step  (11.2%  error  vs  14.8%). 
Speed  of  input  was  p-actical  ly  the  same  under  the  two  methods.  When  two  operators 
checked  each  other’s  translations  before  the  information  was  entered  into  the  data  base, 
error  was  reduced  by  one  third  (10.3%  vs  15.7%) ,  but  the  procedure  took  about  one-third 
more  time  (6.81  min.  vs  4.98  min.).  Either  procedure  was  an  improvement  in  accuracy 
over  the  work  method  of  having  the  message  translated  onto  a  paper  format  by  one 
"action  officer"  and  then  having  a  UIOD  operator  copy  the  format  on  the  CRT  and  then 
enter  the  format  into  the  data  base. 


Utilization  of  Findings: 

The  present  research  strongly  suggests  that  incoming  messages  should  be  translated 
directly  on  the  CRT  screen.  Direct  CRT  input  would  reduce  error  while  eliminating  paper 
formats  and  need  for  the  UIOD  operator  to  transcribe  the  paper  formats  on  a  CRT  screen 
--  a  considerable  saving  in  effort  and  materials.  While  verification  by  a  second  operator 
substantially  reduces  the  number  of  errors  entering  the  system,  a  tradeoff  against  time 
and  manpower  must  be  reckoned  with.  Present  findings  suggest  that,  when  time  and  per¬ 
sonnel  permit,  messages  should  be  verified  for  consistency  before  the  information  is 
entered  into  the  data  base. 
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BACKGROUND 

In  any  semi -automated  system  for  processing  information,  the  point 
of  entry  into  the  automated  subsystem  is  crucial.  It  is  at  this  point 
that  a  translation  process  occurs,  the  end  result  of  which  is  the  input 
to  the  automated  subsystem.  Thus,  in  any  semi-automated  system,  steps 
must  be  taken  to  insure  that  this  translation  is  optimal  with  respect  to 
the  criteria  of  interest  in  the  system. 

The  Army  Tactical  Operations  System  (TOS)  is  one  example  of  a  semi- 
automated  system.  Free-text  messages  arrive  at  the  tactical  operations 
center  and  must  be  transformed  into  appropriate  TOS  codes  before  the  in¬ 
formation  can  be  entered  into  the  computer's  data  base.  In  this  system, 
the  critical  criteria  are  accuracy  and  speed.  The  Army  is  therefore 
interested  in  optimizing  the  process  in  terms  of  having  the  free-text 
message  translated  into  appropriate  and  acceptable  computer  terms  as  ac¬ 
curately  and  rapidly  as  possible. 

There  are  several  potential  subsystem  configurations  that  could  be 
designed  to  deal  with  the  information  translation  and  input  problem. 
Design  of  future  Army  TOS  such  as  the  Army-wide  TOS  require  research- 
based  data  on  the  relative  effectiveness  of  such  configurations  in  order 
to  evaluate  the  speed -accuracy  tradeoffs  within  a  variety  of  potential 
subsystem  configurations.  However,  recent  reviews  by  Mayer  (1)  and 
Shackel  (2)  indicate  that  few  experiments  have  focused  on  man-computer 
input  problems. 

One  potential  configuration  was  incorporated  into  the  7th  Army  TOS. 
However,  it  is  difficult  to  evaluate  the  effectiveness  of  this  configura¬ 
tion  in  the  absence  of  comparable  alternatives.  In  the  TOS  configura¬ 
tion,  one  man  (usually  the  action  officer)  translates  the  free-text  mes¬ 
sage  into  the  appropriate  TOS  codes  on  a  paper  format.  He  then  hands 
this  hard  copy  to  a  second  man,  the  operator  of  the  user  input/output  de¬ 
vice  (UIOD),  who  calls  up  the  appropriate  blank  format  on  his  CRT  screen. 
The  UIOD  operator  then  transcribes  the  codes  from  the  paper  format  on  the 
screen. 


1  Mayer,  S.  R.  Trends  in  human  factors  research  for  military  informa¬ 
tion  systems.  Human  Factors,  1970 >  12>  177 -186. 
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Shackel,  B.  Man-computer  interaction--The  contribution  of  the 
human  sciences.  Ergonomics,  1969 >  ifl>  4B5"^99- 


Certain  questions  may  be  raised  concerning  the  process  just  described. 
The  first  is  the  need  for  paper  formats.  Paper  formats  must  be  preprinted 
and  represent  a  considerable  amount  of  material  that  must  be  transported 
with  the  TOS  equipment  wherever  it  moves.  Also,  the  hard  copy  record 
which  results  is  not  particularly  helpful,  since  the  message  log  provides 
a  manual  back-up.  Finally,  there  is  the  administrative  problem  of  handling 
the  completed  formats  which  contain  classified  information.  Disposal  of 
used  forms  by  burning  might  divulge  the  location  of  the  command  post. 

Wh3t  effect  would  elimination  of  paper  formats  have  on  the  accuracy  and 
speed  of  data  input? 

A  second  question  is  the  potential  value  of  introducing  error  check¬ 
ing  (verification)  into  the  input  procedure.  Does  verification  result  in 
a  significant  decrease  iu  the  number  of  errors  entering  the  system  over 
the  number  entering  without  verification?  If  so,  how  much  input  time  is 
added  by  error  checking?  Is  the  cost  in  time  worth  the  payoff  in 
accuracy? 

The  present  experiment  addressed  itself  to  these  questions.  The 
experimental  task  was  the  translation  of  free-text  messages  into  computer- 
acceptable  terminology.  Table  1  indicates  the  four  conditions  tested. 

In  the  off-line  preparation  mode  (I  and  II),  the  message  was  first  trans¬ 
lated  onto  a  paper  format  worksheet  prior  to  its  transcription  onto  the 
CRT  screen  and  entry  into  the  system.  In  the  on-line  preparation  mode 
(III  and  IV),  the  message  was  translated  directly  on  the  CRT  screen,  by¬ 
passing  the  step  of  first  preparing  a  paper  format.  Data  on  accuracy 
and  speed  of  information  translation  and  transcription  in  the  on-line 
conditions  were  obtained  for  comparison  with  the  off-line  conditions  in 
which  the  message  was  first  prepared  on  a  paper  format.  This  comparison 
determined  what  change  in  performance  could  be  expected  from  the  elimi¬ 
nation  of  paper  formats. 


Table  1 

TWO  BY  TWO  CONTINGENCY  TABLE  SHOWING  EXPERIMENTAL 
CONDITIONS  I -IV 


Data 

Verification 

Preparation  Mode 

Unverified 

Verified 

Off-Line 

I 

II 

On-Line 


III 


IV 


In  the  unverified  conditions  (I  and  III),  one  man  translated  the 
message  into  acceptable  codes  and  entered  the  information  into  the  data 
base  without  formal  error  check.  These  conditions  were  compared  with 
verified  conditions  ( II  and  IV),  in  which  two  men  translated  the  same 
message  and  verified  their  answers  before  entering  the  information  into 
the  data  base.  This  comparison  permitted  an  assessment  of  the  effect  of 
input  verification  on  system  error  and  message  input  time. 


METHOD 


Subjects 

Sixty  subjects,  randomly  assigned  to  four  groups,  served  in  the  ex¬ 
periment.  Three  groups  (I,  II,  III  in  Table  1)  consisted  of  12  subjects 
each  and  the  fourth  (IV)  consisted  of  24  subjects.  Subjects  were  enlisted 
men  enrolled  at  the  U.  S.  Army  Military  Academy  Prep  School.  Since  a 
large  percentage  of  the  subjects  qualified  for  admittance  to  West  Point, 
it  was  judged  chat  these  subjects  were  apt  candidates  for  serving  as 
action  officers  in  the  experimental  task.  Each  subject  received  a  sheet 
of  paper  explaining  the  specific  procedure  he  was  to  follow.  The  proce¬ 
dure  varied  according  to  the  experimental  condition  to  which  the  subject 
was  assigned.  Six  subjects  served  in  each  experimental  session,  one 
subject  per  station.  Four  of  the  six  stations  are  shown  in  Figure  1. 

The  two  subjects  on  the  left  were  monitored  by  one  experimenter,  the  two 
on  the  right  by  a  second  and  the  two  in  the  back  (not  shown  in  Figure  l) 
were  monitored  by  a  third  experimenter. 


Procedure 

Each  subject  was  issued  a  notebook  containing  instructions  for 
translating  the  free-text  messages  onto  appropriate  formats.  The  in¬ 
structions  were  divided  into  two  sections:  One  section  contained  instruc¬ 
tions  for  entries  common  to  most  of  the  formats.  The  second  section  con¬ 
sisted  of  five  subsections  each  containing  instructions  for  completing 
the  specific  format  on  which  the  subject  happened  to  be  working.  A  sample 
format  was  provided  for  each  type  of  format.  The  subject  first  consulted 
the  general  instructions  section  and  then  turned  to  the  instructions  sec¬ 
tion  for  completing  the  specific  format  assigned  to  him  (UA1,  UA6,  UE2, 
UF2,  UJ2) .  A  sample  of  the  instructions  (UA6)  appears  in  the  Appendix,^ 

The  procedures  for  the  four  experimental  conditions  were  as  follows: 

I.  Off-line,  unverified .  In  this  condition,  the  subject  first 
translated  each  message  onto  a  paper  format.  He  then  called  up  the  ap¬ 
propriate  format  on  his  CRT  screen  and  filled  it  out,  copying  from  his 
worksheet . 


^The  material  in  the  Appendix  consists  of  excerpts  from  a  manual  pre¬ 
pared  by  the  Bunker-Ramo  Corporation  for  the  7th  Army  TOS  Development 
Group,  under  contract  to  the  Computer  Systems  Command. 
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IT.  Off-line,  verified.  Subjects  worked  in  pairs.  Each  subject 
translated  each  message  onto  a  paper  format.  The  pair  then  exchanged 
paper  formats  and  each  called  up  the  appropriate  format  on  his  CRT 
screen  and  copied  from  the  other's  paper.  The  pair  then  compared 
answers,  recorded  any  differences  on  a  sheet  provided,  resolved  the  dif¬ 
ferences,  and  entered  the  consensus  version  into  the  data  base. 

III.  On-line,  unverified.  The  subject  translated  the  message 
directly  on  the  CRT  screen. 

IV,  On-line ,  verified.  Subjects  worked  in  pairs.  Each  subject 
translated  the  message  directly  on  the  CRT  screen;  the  two  then  proof¬ 
read  their  screens  to  each  other  and  resolved  any  differences  they 
detected.  They  then  entered  the  consensus  version  into  the  data  base. 


Figure  1.  "Action  Officer"  subjects  manning  four  of  the  six  stations  in  tne  experiment. 
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The  original  number  of  messages  to  be  completed  was  50--10  different 
messages  for  each  of  5  different  types  of  format.  Forty-eight  random 
orderings  of  the  50  messages  were  prepared.  Subsequent  to  the  message 
ordering,  pilot  data  were  collected  on  four  subjects.  These  data  indi¬ 
cated  that  50  messages  would  require  more  time  to  complete  than  could  be 
afforded  by  the  subjects.  The  number  of  messages  was  therefore  trimmed 
to  55  by  deleting  the  last  three  messages  for  each  of  the  five  format 
types.  Thus,  while  50  different  messages  were  used  in  the  experiment no 
subject  received  more  than  55*  Within  each  format  type,  messages  were 
judged  equal  in  difficulty.  Some  subjects  did  not  have  time  to  complete 
all  55  messages.  Chi  square  analyses  of  omitted  messages  indicated  that 
an  equal  number  of  format  types  were  omitted  within  each  condition. 


Stimulus  Materials  and  Apparatus 

The  following  stimulus  materials  and  apparatus  were  used: 

Free-text  messages  typed  on  5"  x  8"  sheets  of  paper 

Paper  formats 

CRTs 

Instructions  notebooks 

A  message  similar  to  the  ones  provided  the  subjects  is  shown  in 
Figure  2.  The  appropriately  completed  message  worksheet  appears  in 
Figure  5*  Figure  4  shows  a  subject  filling  out  the  format  on  his  CRT 
screen. 


Independent  Variables 

The  experimental  design  was  a  2  x  2  factorial  with  12  subjects  in 
5  cells  (I,  II,  III  in  Table  1)  and  12  pairs  in  cell  IV. 

The  two  independent  variables  were  preparation  mode  and  verification. 
each  at  two  levels.  For  the  preparation  mode  variable,  levels  were  off¬ 
line  versus  on-line,  referring  to  the  piesence  and  absence  of  paper 
formats.  Verified  versus  unverified  were  the  two  levels  of  the  second 
variable. 


Dependent  Variables 

The  major  dependent  variables  were  1)  accuracy  as  determined  by  the 
error  rate  for  each  message  completed  by  the  subject  and  2)  speed, 
measured  as  the  time  taken  to  translate  the  message  and  enter  data  into 
the  system. 

Accuracy.  The  measure  of  accuracy  was  an  error  score.  For  each 
format  completed  by  a  subject,  the  number  of  errors  made  was  divided  by 
the  total  number  of  entries  for  that  format.  Thus,  if  the  subjc -t  made 
three  errors  and  there  was  total  of  11  entries  on  a  particular  format, 
his  error  score  would  be  5/H  or  27$.  A  mean  error  score  for  a  subject 
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was  computed  by  summing  the  individual  error  scores  for  each  format  com¬ 
pleted  and  dividing  by  the  total  number  of  formats  completed.  In  the  on¬ 
line  verified  condition,  error  scores  were  the  means  of  the  pair  of  sub¬ 
jects.  This  procedure  resulted  in  IP  mean  error  scores  for  each  of  the 
four  conditions.  These  scores  were  entered  into  an  analysis  of  variance. 

Time  Score .  The  time  score  for  each  message  was  the  total  time  to 
translate  and  enter  the  message  into  the  system  including,  where  appro¬ 
priate,  time  to  complete  the  paper  format,  time  to  input  the  information 
on  the  CRT  screen,  and  time  to  verify  the  data. 


RESULTS 


Accuracy 

Results  of  the  analysis  of  variance  on  the  error  scores  are  pre¬ 
sented  in  Table  2.  They  are  quite  straightforward.  First,  for  condi¬ 
tions  in  which  the  message  was  input  directly  on  the  CRT^  the  error  rate 
was  significantly  less  than  when  the  message  content  was  first  trans¬ 
lated  onto  paper  worksheets  and  then  entered  on  the  CRT.  The  mean  error 
rate  for  direct  CRT  input  was  11.2$  while  that  for  worksheet  method  of 
preparation  was  14.8$.  As  indicated  in  Table  2f  this  difference  yields 
an  F  (l?44df)  of  7*60,  which  is  significant  at  the  .01  level.  The  second 
result  that  clearly  emerges  is  that  the  error  rate  under  the  verified 
input  conditions  was  significantly  less  than  under  the  unverified  input 
conditions . 


UA6 


ROUTINE,  UNCLASSIFIED  REQUEST  FROM  VXG5  FOR  A  REPORT  OF 
ALL  MISSILE  BNS  ORGANIZED  AFTER  130700ZJAN70 .  DON'T  REPORT 
ANYTHING  FROM  VXG3  OR  THAT  WAS  EFFECTIVE  BEFORE  150700ZJAN70 . 
SEND  ALSO  TO  CMGJ.  DON'T  REPORT  EXISTING  UNITS.  VALID  UNTIL 
CANCELLED . 
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Figure  2.  Sample  message 


FRIENDLY  UNIT  INFORMATION 


Figure  3.  Example  of  completed  format  for  message  in  Figure  2 


Figure  4.  "Action  Officer"  subject  completing  format  on  CRT. 


The  mean  error  rate  for  verified  inputs  was  10. 3$  while  that  for  the 
unverified  entries  was  15.7$*  The  differences  yielded  an  F  (l,44df)  of 
17.48  (Table  2),  significant  at  the  .001  level. 


Time  Score 

Profiles  of  message  input  time  indicated  that  the  first  two  or  three 
messages  required  extremely  long  preparation  time,  presumably  while  sub¬ 
jects  were  learning  the  fundamentals  of  translating  the  messages  onto 
formats.  To  reduce  the  effect  of  the  early  time  scores,  median  time 
scores  were  analyzed.  For  the  on-line  verified  condition,  the  median 
time  scores  for  each  pair  of  subjects  were  averaged.  Table  3  shows  the 
results  of  an  analysis  of  variance  of  the  median  time  scores.  The  re¬ 
sults  indicated  no  difference  in  the  time  required  to  input  directly  on 
the  CRT  compared  to  first  filling  out  a  paper  format.  The  average  median 
time  for  these  conditions  was  5*88  minutes  and  5*90  minutes,  respectively. 
However,  time  scores  were  significantly  higher  in  the  verified  than  in 
the  unverified  input  conditions.  The  average  median  time  for  verified 
inputs  was  6.81  minutes,  for  unverified  inputs  4.98  minutes.  This  dif¬ 
ference  yielded  an  F  (l,44df)  of  27.46,  significant  at  the  .001  level. 

While  median  time  was  judged  to  be  more  representative  of  time 
taken  to  complete  a  message,  mean  times  were  also  analyzed.  Results 
complemented  those  obtained  using  median  scores. 


Table  2 

ANALYSIS  OF  VARIANCE  SUMMARY  TABLE  ON  MEAN  ERROR  RATES 


“““ 

Source 

SS 

df 

MS 

F 

P 

Preparation  Mode 

155. ie 

1 

155. 16 

7. 60 

.01 

Verification 

356.98 

1 

356.98 

17.48 

.001 

Interaction 

•93 

1 

•93 

-- 

-- 

Error 

898.60 

ii 

20.42 

1411.67 

4-7 

-  9  - 


Table  3 


ANALYSTS  OF  VARIANCE  SUMMARY  TABLE  ON  MEDIAN  TIME  SCORES 


Preparation  Mode 

15 

1 

15 

-- 

-  “ 

Verification 

145530 

1 

145530 

27.46 

.001 

Interaction 

4780 

1 

4780 

-- 

-- 

Error 

23309.9 

44 

5298 

383424 

47 

Discussion 

With  respect  to  the  criterion  of  accuracy,  no  gain  accrues  by  first 
filling  out  a  message  on  a  paper  format  worksheet  and  then  entering  it  on 
the  CRT.  In  fact,  errors  are  significantly  reduced  when  the  message  is 
translated  directly  on  the  CRT  screen,  bypassing  the  paper  format. 
Furthermore,  direct  CRT  input  is  no  more  time-consuming  than  off-line 
preparation. 

In  a  direct  CRT  input  procedure,  an  action  officer  could  format  the 
message  directly  on  the  CRT  screen.  Paper  formats  would  be  eliminated, 
as  well  as  the  need  for  a  UTOD  operator  in  his  present  role  as  tran¬ 
scriber.  Such  a  deletion  would  result  in  considerable  saving  of  material 
and  manpower. 

It  is  legitimate  to  speculate  on  the  attitude  of  the  action  officer 
toward  inputting  directly  on  the  CRT.  He  might  consider  such  a  task 
menial  and  therefore  resent  it.  There  is  some  evidence,  however,  to 
suggest  that  action  officers  would  be  willing  to  use  the  CRTs.  In  a 
recent  -i'.rvey  of  staff  action  officers  and  noncommissioned  officers  who 
had  used  7th  Army  TOS,  Mace  and  Baker  (3)  found  that  82$  of  the  respon¬ 
dents  believed  the  action  officer  capable  of  operating  the  UIOD. 

Error  checking  was  also  found  to  significantly  reduce  the  number  of 
errors  entering  the  system.  Error  checking  in  the  present  experiment 
reduced  the  error  rate  by  one-third.  In  this  case;,  however,  there  is  a 
tradeoff  to  be  reckoned  with:  While  verification  can  reduce  the  error 
rate  substantially,  it  does  so  at  a  cost  of  time.  Verified  data  input 
required  approximately  two  minutes  per  message  Longer  than  unverified 
inputs.  Thus,  the  one-third  reduction  in  error  rate  was  accompanied  by 
approximately  a  one-third  increase  in  time. 

3 Mace,  D.  J.  and  J.  D.  Baker.  An  assessment  of  the  impact  of  automation 
on  field  Army  users:  V.  Post  field  exercise  findings.  BESRL  Technical 
Research  Note  (in  preparation). 


SUPPLEMENTARY  ANALYSIS 


«**•*:**.; r^9V  iftVin  V 


In  the  7th  Army  TOS,  an  action  officer  translated  a  message  onto  a 
paper  format  and  handed  it  to  a  UIOD  operator  to  log  into  the  system. 

Such  a  procedure  involves  paper  formats  and  unverified  data  inputs,  both 
of  which  the  present  research  has  shewn  to  be  sources  of  error.  Another 
potential  source  of  error  existed  in  the  7th  Army  TOS  procedure.  The 
UIOD  operator  had  to  copy  from  a  worksheet  prepared  by  someone  else. 
Judging  from  a  previous  BESRL  experiment  (4),  procedures  involving  entry 
of  photointerpretation  data  into  a  computer  without  a  transcriber's  act¬ 
ing  as  intermediary  had  the  most  potential  for  efficient  man-computer 
communication. 

In  order  to  determine  what  error  rate  would  be  expected  if  the  TOS 
procedure  had  been  employed  in  the  present  experiment,  additional  data 
were  collected  using  the  TOS  procedure.  One  man  acting  as  the  action 
officer  translated  the  free-text  message  onto  a  paper  format  and  handed 
it  to  a  UIOD  operator  who  called  up  the  appropriate  format  on  his  CRT, 
copied  from  the  paper  format  on  the  CRT,  and  entered  the  completed  for¬ 
mat  into  the  data  base.  Data  were  collected  for  twelve  such  pairs  of 
operators.  The  subjects  were  enlisted  men  who  had  recently  graduated 
from  the  Photo  Intelligence  School  at  Fort  Kolabird,  Maryland.  The 
error  rate  for  this  condition  is  shown  in  Table  4,  along  with  comparable 
error  rates  for  the  other  four  conditions.  The  TOS  condition  yielded  the 
highest  error  rate,  21.3$.^  In  order  to  obtain  an  estimate  of  the  num¬ 
ber  of  copying  errors  made  by  the  subject  acting  as  the  UIOD  operator  in 
the  TOS  condition,  the  message  worksheets  filled  out  by  the  subjects  act¬ 
ing  as  action  officers  were  scored.  The  mean  error  rate  for  the  TOS 
worksheets  was  Thus,  copying  errors  added  7.7$  to  the  error  rate, 

increasing  the  error  rate  by  57^. 


4Root,  R.  T.,  D.  Waugh,  K.  Hewitt,  and  J.  Donoghue.  An  Analysis  of 
interpreter-computer  reporting  techniques.  BESRL  Technical  Research 
Note  170.  June  1966.  (AD  645  295). 

^ In  the  7th  Army  TOS,  for  example,  the  computer  performed  an  error  check 
on  "mandatory"  entries  through  an  edit  and  validate  subroutine.  Any 
incorrect  mandatory  entry  would  result  in  the  entire  message's  being  re¬ 
turned  for  correction  and  reentry.  The  present  experimental  conditions 
did  not  involve  machine  error  checking.  The  programming  required  to 
incorporate  this  feature  into  the  present  study  would  have  been  pro¬ 
hibitive  in  time  and  costs.  Also,  it  was  considered  better  to  make 
the  commission  of  such  errors  irrevocable  in  order  to  get  clear-cut 
error  rates  per  trial.  Machine  verification  would  perhaps  have 
lowered  error  rate  across  all  the  experimental  conditions.  Error  for 
the  TOS  condition  cv0u]^  therefore  be  considered  with  respect  to  other 
conditions  in  the  present  experiment. 
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Table  4 


COMPARISON  OF  TOS  ERROR  RATE  WITH  THAT  OBTAINED 
UNDER  EXPERIMENTAL  CONDITIONS 


TOS 

Off-line 

Unverified 

Off-line 

Verified 

On-line 

Unverified 

On-line 

Verified 

21.3$ 

17.6$ 

11.9$ 

13.7$ 

8.6$ 

The  obtained  error  rate  of  13 -6%  is  in  close  agreement  with  the 
error  rate  of  13*7%  obtained  in  the  on-line  unverified  condition.  Thus, 
13*14$:  appears  to  be  a  stable  baseline  estimate  of  the  error  rate  for 
translating  a  free-text  message  onto  a  computer-acceptable  format.  This 
estimate  includes  only  errors  committed  by  translating  the  message  con¬ 
tents  onto  the  appropriate  format. 

The  problem  of  selecting  the  appropriate  format  was  not  involved  in 
the  present  experiment  (subjects  were  informed  of  the  correct  format), 
but  is  dealt  with  elsewhere  by  Baker  and  his  associates  (5),  who  found  an 
error  rate  of  22$  in  selecting  the  appropriate  format.  Considered  to¬ 
gether,  these  estimates  indicate  the  error  rates  that  can  be  expected 
from  the  processes  of  format  selection  and  completion  in  a  TOS-type 
system  in  which  there  is  neither  man  nor  machine  verification. 

Further  analyses  were  conducted  on  the  error  data,  categorizing 
errors  into  different  types — errors  of  omission,  commission,  copying, 
etc.  These  analyses  revealed  that,  for  the  most  part,  the  error-reducing 
effects  of  on-line  inputting  and  verification  consistently  reduce  errors 
of  all  types.  The  analyses  confirmed  the  above-mentioned  finding  that 
most  of  the  increase  in  error  in  the  TOS  condition  comes  from  copying 
errors. 

Research  directed  toward  the  development  of  an  input  error  taxonomy 
is  in  progress.  The  taxonomy  will  be  incorporated  into  a  general  model 
of  human  performance  in  information  systems  [see  Baker  (6)]  to  permit 
evaluation  of  the  impact  of  each  input  error  type  at  every  stage  of 


^  Baker,  J.  D.,  D.  J.  Mace,  and  J.  M.  McKendry.  fhe  transform  opera¬ 
tion  in  TOS:  Assessment  of  the  human  components.  BESRL  Technical 
Research  Note  212.  August  1969.  (AD  697  7I6) 

Baker,  J.  D.  Quantitative  modeling  of  human  performance  in  information 
systems.  Ergonomics,  1971?  13.,  645-664. 
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system  throughput,  as  well  as  on  overall  system  performance.  Once  criti¬ 
cal  error  types  have  been  identified,  appropriate  remedial  steps  may  be 
taken.  At  the  present  time,  it  is  hypothesized  that  there  are  critical 
input  errors  which  would  not  be  detected  by  the  system  software  (e.g., 
inverted  digits  in  a  map  coordinate).  The  taxonomy,  when  developed,  and 
the  resulting  implications  for  system  design  will  be  published  in  a 
future  BESRL  report . 


CONCLUSIONS 


Admittedly,  not  all  possible  procedures  were  sampled.  As  automated 
information  processing  systems  evolve,  other  procedures  may  emerge  that 
are  not  evident  at  this  time.  The  approach  taken  here  was  to  seek  gener¬ 
al  izable  results  for  design  consideration  over  many  systems  (TOS,  TACF1RE, 
IBCC,  etc.).  Specific  detailed  studies  of  various  input  techniques 
should  be  subsequently  evaluated  in  the  context  of  system  test  and 
evaluation  efforts. 

In  the  present  experiment,  data  input  accuracy  was  significantly 
increased  when  free-text  messages  were  translated  directly  on  the  CRT 
screen  rather  than  first  filled  out  on  paper  formats.  The  implication 
for  improved  system  performance  based  on  this  finding  is  clear:  Incoming 
messages  should  be  translated  directly  on  the  CRT  screen.  This  procedure 
would  eliminate  the  need  for  paper  formats. 

Accuracy  of  data  input  was  also  significantly  increased  when  the 
data  were  verified  before  being  entered  into  the  system.  Such  a  proce¬ 
dure  did,  however,  significantly  increase  data  input  time,  and  is  there¬ 
fore  recommended  only  if  time  and  personnel  permit.  In  that  case,  each 
message  should  be  formatted  independently  by  two  action  officers  and 
verified  for  consistency  before  the  information  is  entered  into  the  data 
base. 

Results  with  a  procedure  similar  to  that  used  in  the  7th  Army  TOS, 
in  which  a  UIOD  operator  transcribes  paper  formats  on  the  CRT,  indicated 
that  copying  errors  introduced  by  the  UIOD  operator  increases  the  error 
rate  from  13.6$  to  21.3$.  This  finding  lends  further  support  to  the 
conclusion  that  the  transcribing  step  should  be  dropped  from  the  input 
procedure . 
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APPENDIX 


INSTRUCTIONS  TO  SUBJECTS  FOR  TRANSLATING  FREE-TEXT  MESSAGES 
INTO  APPROPRIATE  TOS  CODES 
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GENERAL  INSTRUCTIONS 


In  most  problems,  you  will  enter  the  follow' ng  items  of 
information: 

PRECEDENCE  -  Establishes  the  priority  of  the  message. 

If  the  precedence  ’ 


Flash 

Immediate 

Priority 

Routine 

HARD  COPY  -  Always  enter  a  Y 


you  enter 

F 

I 

F 

R 


ORIGIN  -  Identifies  the  originator  of  the  message.  ORIGIN  is 
a  combination  of  a  one-character  organization  code,  a 
one-character  headquarters  code,  and  a  two-character 
staff  element  code,  in  that  order. 

EXAMPLE  -  If  the  originator  of  the  message  were  the 

Third  Armored  Division,  G3>  Main,  you  would 
enter  SMG3  beside  ORIGIN.  Each  message  will 
contain  the  code  identification  (such  as 
SMG3)  to  be  entered  beside  ORIGIN. 

SCTY  -  Identifies  the  security  classification  of  a  message. 


If  the  security  is 


Unclassified 

Confidential 

Secret 


you  enter 

UNCLAS 

CONF 

SECRET 


RESTR  -  Restriction  will  appear  only  on  certain  messages. 

If  the  restriction  is  you  enter 

Originator  only  A 

Not  releasable  to 

foreign  nationals  B 

Not  releasable  to  foreign 

nationals  except  GY  C 

Restricted  data  D 

Formerly  restricted  data  E 

NATO  F 

ATOMAL  G 

Division  and  below  only  K 

Division  and  above  only  L 

Corps  and  below  only  M 

Corps  and  above  only  N 
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PRECEDING  PA6E  BLANK 


RESTP  (cont’d) 


If  the  restriction  Is  you  enter 


Army  and  below  only  0 
Amy  only  P 
Intelligence  channels  only  Q 
Logistics  channels  only  R 
Operations  channels  only  S 
Personnel  channels  only  T 
Upward  command  channels  only  U 


Downward  Command  channels  only  V 

UNIT  -  The  military  unit  identification  or  the  assigned  switchboard 
designation  of  the  unit. 

EXAMPLE  -  If  the  data  concerned  the  Third  Battalion,  Seventy- 
Sixth  Armored  unit,  you  would  enter  3-BN-76-ARMD 
beside  UNIT.  Each  message  will  contain  the  code 
identification  ('such  as  3-BN-76-ARMD)  to  be  entered 
beside  UNIT. 

To  complete  the  remainder  of  the  message  format,  refer  to  the  special 
instructions  which  are  specific  to  each  format. 
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INSTRUCTIONS  FOR  ONE  SPECIFIC  TYPE  OF  FORMAT 


UA6  -  Friendly  Unit  Task  Organization/Task  Force  SRI  Establish 


Purpose  -  The  Task  Organization/Task  Force  SRI  Establish  message  is 
used  to  establish  a  standing  request  for  information  (SRl) 
on  selected  task  organization  or  task  force  data,  data 
change,  and  data  delete  messages. 

EXAMPLE  MESSAGE  -  An  example  message  is  shown  below.  The  resulting 
TOS  message  composed  on  the  appropriate  format  is 
illustrated  in  Figure  UA6. 

EXAMPLE  -  SMG3  submits  a  routine  unclassified  request  for  a 
report  on  any  combat  unit  of  battalion  echelon  or 
higher  that  is  being  organized  or  changed  from  now 
until  31  August  1967  at  2400  hours  when  this  request 
is  cancelled.  Do  not  report  changes  input  by  "SMG3", 
and  do  not  report  the  existing  task  organizations. 

FORMAT  -  Does  not  apply  to  any  of  the  messages  you  will  process. 

VALID-TO  -  This  information  item  specifies  a  time  until  which  this 

standing  request  for  information  (SRI)  will  remain  valid. 

The  entries  for  this  information  item  are  a  date-time 
group  specifying  the  time  of  the  SRl's  deletion,  or  the 
word  OPEN. 

EXAMPLE  1  -  VALID -TO/3 12400 ZAUG67 ;  The  standing  request  for 

information  described  in  this  message  is  scheduled 
to  be  dropped  from  the  system  at  2400  Zulu  on  31 
August  1967. 

EXAMPLE  2  -  VALID  TO/OPEN;  The  standing  request  for  information 

described  in  this  message  will  remain  in  the  TOS  until 
recipients  specifically  request  its  deletion. 

INHIBIT-OWN  -  Allowable  entries  are  YES  and  NO.  If  the  originator  of 
the  SRI  enters  a  YES,  he  will  not  receive  future  output 
reports  distributed  in  response  to  this  SRI  if  the  out¬ 
put  resulted  from  one  of  the  originator's  own  input 
messages.  A  NO  entry  will  not  suppress  future  output 
message  dissemination. 

EXAMPLE:  INHIBIT-OWN/YES ;  The  originator  of  this  message  will 
not  receive  output  distributed  in  response  to  his  own 
input  messages. 

QUERY  -  This  information  item  is  used  to  request  an  initial  interro¬ 
gation  of  TOS  stored  data.  Allowable  entries  are  YES  and  NO. 

A  YES  entry  will  cause  the  system  to  immediately  query  the 
stored  data  and  send  to  the  SRI  originator  a  query  response 
output  message  listing  units  satisfying  the  retrieval  criteria. 
When  NO  is  entered,  the  SRI  will  not  query  the  stored  data. 
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FRIENDLY  UNIT  INFORMATION _ 

TASK  ORGANIZATION/ TASK  FORCE  SRI  ESTABLISH) 


Figure  UA6.  Example  of  a  Task  Organizational  Task  Force  SRI  Establish  Message 


EXAMPLE  -  QUERY/NO;  An  initial  interrogation  of  the  TOS  will 
not  be  performed. 

RECIPIENTS  -  This  information  item  is  used  to  designate  up  to  seven 
recipients,  other  than  the  originator,  of  outputs  in 
response  to  the  SRI. 

EXAMPLE  -  RECIPIENTS /CMG3/CMFS/CMG2/  /  /  /  /;  In  addition 

to  the  initiator  of  the  SRI,  G3,  Fire  Support,  and  G2 
of  Cyclone,  Main  are  scheduled  to  receive  copies  of 
any  output  messages  routed  as  a  result  of  the  SRI. 

UNIT  -  The  military  identification  or  the  assigned  switchboard  desig 
nator  of  the  unit  whose  data  are  requested  from  the  TOS. 

EXAMPLE  -  UNIT/12-BN-87-ARTY:  Data  on  the  Twelfth  Battalion, 
Eighty-Seventh  Artillery  are  requested  from  the  TOS. 

TF-NAME  -  The  code  name  assigned  to  a  task  force  for  purposes  of 
identification. 

EXAMPLE-  TF-NAME /UPSTART;  Data  are  requested  from  the  TOS  on 
the  task  force  UPSTART. 

ECHELON  -  This  information  item  may  contain  two  entries.  The  first 
entry  is  a  relational  operator.  Select  the  appropriate 
entry  from  the  following  table: 


Relational -Operator 


Entry 

equal  to 

EQUAL 

less  than 

LESS 

more  than 

MORE 

no  more  than 

NOMORE 

no  less  than 

NOLESS 

The  second  entry,  when  used,  consists  of  a  standard  military 
echelon  code.  The  two  entries  in  conjunction  define  a  range 
of  echelons  of  units  whose  data  are  to  be  retrieved  from  the 
TOS. 

EXAMPLE -ECHELON/NOLESS /BN ;  Data  are  requested  only  on  units  of 
battalion  or  higher  echelon. 

TYPE  -  The  originator  may  use  this  information  item  to  specify- 
military  type  of  unit. 

EXAMPLE  -  TYPE/PERSH;  Only  data  on  pershing  missile  units  are 
requested. 

BRANCH  -  The  military  branch  of  the  unit  whose  data  are  requested  from 
the  TOS . 

EXAMPLE  -  BRANCH/ARTY :  Data  are  requested  only  on  artillery  units. 
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CATEGORY  •  The  military  category  of  the  unit  whose  data  are 
requested  from  the  TOS. 

EXAMPLE  -  CATEGORY/C BT;  Data  are  requested  from  TOS  only  on 
units  in  the  combat  category. 

NATION  -  The  nationality  of  the  unit  whose  data  are  requested  from 
the  TOS.  In  all  messages,  the  nationality  is  the  United 
States.  Thus,  the  letters  US  should  ba  placed  beside 

NATION. 

EXAMPLE  -  NATION/US;  Data  concerning  a  United  States  unit  are 
requested  from  the  TOS. 

SUBOR-TYPE  -  leave  blank 

SUBOR-TO  -  leave  blank 

TIME-FRAME  -  This  information  item  contains  two  entries  and  is 
used  to  specify  the  beginning  and  ending  of  a  time 

period . 

EXAMPLE  -  TTMF. -FRAME/FROM/1  EQ800ZSEP67/T0/ - - b 

that  a  report  on  all  data  messages  having  an  effective 
time  after  0800  Zulu  on  15  September  I96T  are  to  be  re¬ 
ported. 

ENTERED-BY  -  leave  blank 
CLASSIFIED  -  leave  blank 
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